The idea of my new project is to retrieve the data from job-searching websites and provide better filtering, searching and visualization.

I think for the most people who read this, searching for a job in Internet is a pretty common activity. Even if you are not going to change job right now, it might be quite interesting to know what skills are currently the most valuable for your specialization and what is going on on the Global labor market.

What’s wrong with job searching sites

To search for a job you will most likely go to some job-searching site or a business social network (btw, have you already add me to your connection list at linkedin? 😉 ) and start to make search requests in some web interface, then you look into results and decide which of the jobs you find is best suitable for you.

This works well when you are searching for some job titles in a particular location. But if you are looking for something rare and want to see what is happening globally in your sector for the labor market, filtering can be a tricky task:

• If you make some general request, you will get lots of results that won’t meet your needs.
• If you add some more keywords to the search request you will easily miss something. Just because the vacancies can be described in a very different manner and may not, for example, contain the name of your favorite programming language or framework. On the other side, it might be used in some vacancies, as one of the many additional skills, which is also rubbish.

At least you will need to set

• some locations that you like and some location that you don’t like
• some keywords that should exist in the description of the vacancies and some that shouldn’t

To do this, you will need much more advanced tools than those that are currently exist on job searching websites. Because in fact these sites are not for the individuals, who are searching for a job, but for the companies that are searching for the staff. They take money from the companies to post the information about vacancies an work on the features for these companies. It’s pretty natural strategy, because it’s much easier to monetize.

How Open Positioner works

So the most realistic way to do advanced search is to export the vacancies from job-searching sites and then to process them with some scripts. All the code consists of three parts:

1. Parsing. These part of code makes the search request on the sites and it gets the data about the vacancies: company name, title, location and URL. It’s the most sensitive part because nearly all job searching websites don’t like such automated actions, they try to detect it and ban.
2. Processing location data. The location in the search results is usually presented in a non-formalized way. Basically it’s name of the city, state and the country. To get the coordinates from these data you need geolocation API, for example I use free service by HERE.
3. Reporting. Finally, when we have all the vacancies in a feed, we can process, group and show them on a map (I use folium module for this).

NB. It’s an open question if it is ethical and legal to gather these data from the job searching sites. In my opinion, if the authentication is not used and terms of usage of the site were not accepted, it’s ok. In this case the parser works on the same conditions as Google indexing robots.

Example: “Vulnerability Management” vacancies published last week

Let’s see for example the “Vulnerability Management” vacancies posted during the last week. There were more than 2,000 vacancies, but the platform allows us to get only latest 1,000. Among these vacancies there were some doubles and I tried to get rid of them automatically, reducing total number to 836. It’s not an easy task because some of this doubled vacancies were published by the employment agencies in a slightly different form. Actually it’s not a big deal and just creates a small error in statistics.

The amount and percentage of open ‘Vulnerability Management’ positions by country:

United States	588	70.3%
United Kingdom	70	8.4%
Europe: Blue Card Zone	53	6.3%
India	26	3.1%
Canada	23	2.8%
Netherlands	16	1.9%
Ireland	16	1.9%
South Africa	12	1.4%
Switzerland	8	1.0%
Germany	8	1.0%

Let’s see the TOP three.

United States

As you can see on the image above, most of the vacancies (more than 70%!) are from US. It certainly depends on the job searching site that you parse. But I also think it’s a fact that most cyber security jobs are currently located in US.

If you are not a citizen and don’t have a green card, you will need a work visa, in most cases H-1B, that should be sponsored by the employer. So, only very big companies can afford to relocate security specialist without much trouble. For Vulnerability Management vacancies the citizenship and security clearance are often mandatory requirements. And it’s quite interesting that most of such vacancies are located on the East coast, probably because most of financial institutions are there and they need VM for compliance.

United Kingdom

The next popular location is UK. It’s a bit easier to get the work permit (Tier 2 General) there.

However, if you are not from the English speaking country you will need to pass IELTS exam. And the company will need to sponsor your Visa. In Ireland requirements are similar, but, as far as I know, they do not require mandatory English exam.

European Union and Blue card

Most of European countries, 25 out of 28 European Union members, excluding Denmark, Ireland and the United Kingdom, participate in Blue Card program.

The visa for Blue Card process is much easier than for UK and US and more comfortable for the companies. Also the card holder can later easily change the work inside of these blue card zone. So I combined all the countries that are participating in these program in one virtual “country”. It also demonstrates how such grouping can be done in Open Positioner.

In conclusion

So it’s pretty much like these: data representation and grouping in a more convenient way. What’s next?

1. I want to group of these vacancies not only by countries but by regions and by the rating of the city.
2. It’ll be great to get the detailed description of the vacancies as well. Then it will be possible to understand automatically what the company needs from a person on this position.
3. It’ll be also great to do something with all these ridiculous forms that should be filled to apply the vacancy. I think it can be automated significantly as well.