Build an antivirus solution with auto-updates

Are you able to threat not having an antivirus on your app?

Let’s say that your system has an add/obtain operate for information. Customers can add them through the online software and in a while, they will determine to make the information accessible to different folks, to allow them to obtain them to their gadgets. 

Additionally, think about that the system’s staff can connect the information to particular customers’ storage instantly by utilizing a CRM, whereas additionally having the ability to browse the storage itself.

Within the AWS world, one doable method could be to introduce an S3 bucket the place information could be saved. With the intention to assist file add and obtain by customers, we are able to use AWS presigned URLs.

Now, what occurs if a buyer uploads a file — purposely or not — that’s contaminated by a trojan?

What unprotected file switch could cause

Whereas AWS retains the file innocent, if somebody downloads the malicious knowledge, they’re in serious trouble… It may trigger a system leak or a service crash and the danger is actual. The typical value of an information breach now stands at $1.07 million (IBM).

There could be an avalanche of penalties when this occurs:

• Reputational injury
• Dropping confidentiality and safety your customers count on
• The potential of exposing the system’s delicate knowledge, if an worker’s system will get contaminated

The perfect method could be to take away that nasty add/obtain performance to stop malware from spreading. 100% security assured!

Since this isn’t a believable possibility… Let’s take into consideration how we are able to construct our personal customized virus scanning resolution which is able to kick out any viruses.

In a world of servers, virus safety is fairly easy

There are just a few open-source, anti-malware toolkit options that you should use to carry out a scan. One among them is ClamAV, an engine for detecting trojans, viruses, malware & different malicious threats. 

Merely working with containers/servers, we might configure such an antivirus engine on the container/machine and let it do the job. 

We wouldn’t even have to fret about updating the virus definitions database. ClamAV already comes with a built-in software referred to as freshclam, which takes care of downloading and updating ClamAV’s official virus signature database.

Nevertheless, as lazy serverless builders who don’t need server administration in any respect, we have now to determine how an antivirus resolution can work on serverless.

Earlier than we begin, I’d prefer to thank the neighborhood

What I like most concerning the software program engineering trade is the neighborhood that likes to share data. 

Educating one another works like a snowball. Somebody shares their latest growth experiences, folks learn them and say “wow, I would like to do that out!”. And so they strive.

Later, they share their very own ideas on the topic, inspiring a wave of different engineers to construct on that work and produce again implausible options! 

I used to be hit by a snowball rolled by the professionals earlier than me. I wish to make it even greater and push it even additional!

Large because of Joseph Sutton whose information to constructing a serverless virus scanner helped me set the foundations of my prolonged resolution. With out the work that he had finished, this wouldn’t be doable ?

Professional-tip: You may need to learn it first to know what we’re doing right here.

I reused what he constructed for a Lambda that scans information because of container photos assist. Then, I added a operate of computerized updates for the virus definition database, presenting some real-life use examples for the app alongside the way in which.

You gained’t discover the method of constructing a Lambda scanner described in-depth, however I’ll summarize what Joseph did, in short, to concentrate on the enhancements I added.

Lambda container photos can assist you create a customized runtime

With Lambda features, we don’t provision or handle any servers. AWS offers us the checklist of supported runtimes we are able to run our code on. We merely decide one and depart the remaining to the cloud supplier. 

Sounds convincing and helpful. Issues get tougher after we need to do one thing customized that AWS’ pre-defined runtimes don’t supply, which we can’t do by utilizing some library or a framework. 

In that case, we want a customized runtime that the Lambda operate can run on to fulfill our subtle wants.

As you may already know from Joseph’s piece, on the finish of 2020, AWS launched assist for Docker photos. We are able to now construct our personal Lambda runtime that comes full of all the things a developer wants.

With a most measurement of 10 GB for a Docker picture, we are able to do actually nice issues. In our case, we simply want the ClamAV antimalware toolkit and AWS node.js Lambda runtime put in, and we ought to be able to go.

To create a Dockerfile, merely seize the one from Joseph’s article (written very well) or attempt to create your personal one. 

Protip: ClamAV has the official picture that you should use so as to not do all the things manually.

One factor I wish to spotlight right here is the half beneath.

Arrange all of the stuff that ClamAV wants. Utilizing the multi-stage Docker construct, import the official AWS Lambda’s Node.js picture that implements the Lambda runtime API.

In case your Lambda wants any dependencies, copy the dependency information (by way of Node.js these could be bundle.json and package-lock.json) and set up them contained in the picture.

A very powerful a part of the Lambda is described beneath.

We name the ClamAV executable to carry out an precise scan. If the error’s standing is the same as 1, which means the file has been detected as contaminated. Then, it’s your alternative tips on how to deal with this situation. Let me present you what I’d do.

How the app handles an contaminated file

I arrange a short lived bucket the place our information could be saved after being uploaded. If the file is contaminated, we transfer it right into a bucket with no entry for customers. Let’s name it a quarantine.

But when the file is wholesome, it strikes ahead to a secure place customers can entry.

Scanning the file for malware

The consumer uploads a file to an S3 bucket referred to as uploaded-files. Due to the AWS S3 occasions, we don’t care concerning the UI or how the add is applied. All we care about is that if a file lands contained in the bucket.

If it does, a Lambda is triggered, and it performs a scan.

If the file is wholesome, we generate an Occasion Bridge occasion to tell customers the file is ok. Our handle-healthy-file Lambda listens for such occasions, so it might probably transfer the file to the healthy-files S3 bucket, the place our customers can entry and obtain them safely.

If the file is malicious, we ship the Occasion Bridge occasion informing that the file has been detected as contaminated.

In keeping with the one duty precept — and so as to not combine enterprise logic with technicalities — I didn’t need the scanner operate to do the rest. It solely scans the information and notifies (doesn’t matter who) if the file is contaminated.

Within the subsequent step, our handle-infected-file operate strikes the file to an remoted S3 bucket referred to as infected-files the place it stays locked with no buyer entry doable.

After the file hits the isolation bucket, Lambda sends one other Occasion Bridge occasion to say the file has been moved into quarantine. Now, the scenario is below management, however there’s extra to do!

Notifying the consumer about malware

The`Notifier Service` within the diagram reveals how the app can inform a safety knowledgeable that they should act after a file is quarantined.

We’ve got a number of notifiers (additionally Lambda features) which inform our system by way of totally different communication channels. 

One Lambda sends a Slack message, one other one creates an alert, and one other one marks who despatched the malicious file within the CRM. By no means go to lunch with them!

Due to AWS Occasion Bridge occasions, we’re versatile in including extra notifiers and eradicating the present ones that aren’t wanted anymore!

Find out how to make ClamAV auto-update the virus definitions database

One factor I used to be lacking in Joseph’s resolution was an replace operate for the virus definitions database. 

As soon as we construct the picture and push it to ECR, it doesn’t get up to date anymore except we redeploy the entire service.

ClamAV does all the things for us because of the built-in freshclam software.

Below the hood, it runs a CRON job which pulls the most recent definitions from a database that’s being up to date each day.

For ClamAV to work with serverless, you must do some tweaking.

Whereas considering how this may be solved, another shiny minds and I got here up with two neat concepts. Let’s examine them and determine which one is a greater match.

Possibility 1: Static container & dynamic storage resolution

Have a look at the diagram beneath to know what thriller is hiding below this title.

I didn’t put the remainder of the method right here in an effort to hold issues clear. What I’ve added are the three additional elements. Counting from the highest, these are:

• virus-definitions S3 bucket
• update-virus-definitions Lambda operate
• Occasion Bridge scheduled occasion (or just a CRON job, since we’re used to calling it that!)

The thought right here is to extract the virus definitions database to some exterior storage, which in our case is an S3 bucket. 

As a result of we extract our database from the picture itself, we are able to introduce a course of that may hold it up to date.

Possibility 1: The circulation

We are able to have the additional Lambda operate referred to as update-virus-definitions which is triggered by an Occasion Bridge occasion below a set frequency.

First, it downloads the ClamAV virus signature database from an official supply. Then, it overrides the previous database with the brand new one.

Now, our Lambda that scans the information (referred to as scan-file) has to first obtain the database from an S3 bucket.

The static container & dynamic storage resolution’s large benefit is that we are able to take away the virus definitions database from our Dockerfile as a result of we transfer it into an S3 bucket.

Because of this, constructing the picture and pushing it to ECR is means quicker, shortening deployment time.

Are you able to see now why I referred to as it a static container and dynamic storage resolution? 

We’ve got a container that may be constructed solely as soon as (static container) and a storage (an S3 bucket) that’s being constantly up to date (dynamic storage).

Possibility 1: Disadvantages

• Earlier than performing an precise scan for a single file, a database of round 300 MB needs to be downloaded first.
  
  Transferring the information from and to an S3 bucket contained in the cloud is quick — however not quick sufficient for this. My time take a look at of a Lambda must obtain and save S3 bucket information gave me 4 seconds you must add to complete scan time.
• Lambda features have a restricted non permanent storage measurement of round 500 MB.
  
  Though AWS Help Middle can elevate some limits for you, this one is a no-go. It’s a tough restrict it’s a must to respect identical to with Lambda’s most execution time, or max RAM.
  
  Because the database weighs 300 MB, that leaves you one thing round 200 MB for an uploaded file that we additionally must obtain to scan it.
  
  Is it a lot? In case your app helps importing information of at most 5 MB, then you may contemplate this resolution. However for file uploads bigger than 200 MB, effectively… that’s a no-no for you.
• It’s not solely concerning the measurement of the scanned information but in addition concerning the database measurement.
  
  Because it’s being up to date 24/7, it should proceed to develop in measurement. For me, the dimensions elevated by 3 MBs between December 2021 and January 2022.
  
  You may say: “it’s not that a lot!”. Effectively, in the event you plan to construct a system that may final for years, ultimately you’ll attain the restrict that Lambda /tmp storage can deal with.

For me, the final drawback utterly cancels this resolution.

Possibility 2: A dynamic container resolution

As beforehand, first take a look at the diagram beneath.

This resolution doesn’t have any storage. The database is certain to the Docker picture itself, identical to within the authentic resolution. 

To maintain the virus definitions database up to date, the app builds, tags, and pushes a brand new Docker picture to the AWS Elastic Container Registry, after which updates our scan-file operate’s code with the final pushed picture.

A GitHub Actions pipeline for the dynamic container

Utilizing GitHub Actions is my most popular CI/CD resolution. Because it’s a quite simple motion, I’m positive it should work with different workflow automation platforms like BitBucket Pipelines, GitLab CI, or Jenkins.

Right here’s a reusable GitHub Actions pipeline that updates the virus definitions database.

What you see above GitHub names a workflow_call (see GitHub documentation). You would examine them to features.

The GitHub Actions pipeline presents a chunk of code that takes some parameters (precisely like a operate) and does one thing for you. 

It’s a extremely helpful means of not duplicating your pipeline code in a scenario when you will have a number of environments you deploy to. The deployment course of most likely would be the similar for every of the environments with totally different parameters. 

So we have now GitHub motion that in a while we’ll apply for various levels (represented by totally different AWS accounts) of our software.

Let’s analyze what’s taking place right here. 

Below inputs and the secrets and techniques sections, you may see the enter parameters. Most of them are apparent.

ecr_repository_name is the place the Docker picture sits, which is required to rebuild the Lambda scanner. function_name is the scan-file Lambda operate, which is the one which the app rebuilds.

The pipeline has two jobs in complete.

First job: Construct, tag, and push the picture to ECR

The primary job is to construct a picture and push it (or to be exact — to override an previous one) to the AWS Elastic Container Registry. 

A very powerful step is named Construct, tag, and push the picture to ECR.

Utilizing atmosphere variables, we outline the ECR registry, the repository title, the picture tag we create for our picture, and the trail to the listing the place our ClamAV Dockerfile sits in.

As you may need observed, a picture tag is customized however stays the identical on a regular basis. 

In my case, I don’t want historic photos tagged in another way, however in the event you do, it’s only a matter of including some customized id to the picture’s tag.

That’s actually all that it does. It makes use of official GitHub and AWS actions to the job – no magic right here!

Second job: Replace the Lambda container picture

The second job relies on the primary one. Earlier than it might probably rebuild the Lambda operate’s code, it wants the brand new Docker picture to be already sitting in ECR. 

Right here, crucial step is known as `Replace operate code`. 

It makes use of an AWS CLI command referred to as aws lambda update-function-code which helps --image-uri and --function-name choices. With them, you may merely go an ECR picture URL you’ve simply created and level it to your operate! 

The following command waits for the operate’s LastUpdateStatus to be Profitable. If the replace fails, we’ll find out about it through the failing GitHub pipeline.

All collectively: The precise GitHub motion

Now, you’ll learn to use the reusable workflow for a selected motion.

Let’s say that the pipeline offered above is on your growth atmosphere. In actual life, you most likely produce other ones for various levels like staging or manufacturing. 

Merely present all of them the variables you outlined beforehand and inform the pipeline that it ought to be run on a schedule. In my case, it’s 8 AM every single day. 

You must also specify workflow_dispatch in order that the pipeline could be triggered manually (by GitHub Actions UI, for instance).

It’s a measure ready simply in case you’d prefer to patch the virus definitions database ad-hoc when some new menacing virus is launched.

Is that this serverless resolution well worth the effort?

There’s additionally the choice of putting in ClamAV software program on an on-prem system or inside a Docker container. 

You may say that by going serverless, we kinda misplaced the largest benefit of Lambda features — delegating managing servers to AWS.

Right here’s why you may discover the serverless means the higher different:

• You continue to don’t handle any servers in any respect!
  
  We solely wrote a customized Dockerfile as a one-time motion. Due to the official AWS Lambda photos, you may import one in your Dockerfile and proceed to depend on the stuff that AWS manages in full!
• Lambda features scale extraordinarily effectively
  
  With servers, you should handle scaling our infrastructure whereas serverless options scale flawlessly out of the field because of AWS’ useful hand.
• The value is a cut price, as now, Lambda invocations value $0.20 per a million requests
  
  Within the case of ECR, you solely pay for storage. When you solely retailer one image-tag pair at a time, you’re faaaar away from reaching $0.01 per 30 days. You gained’t be capable to beat that value with a non-serverless method.